Pulling in new changes from parent #1

caitriona-cloudsmith · 2025-09-19T10:21:41Z

No description provided.

…ute_iter

…d for Ubuntu 24.04.

Specifically, toStartOfMinute, toStartOfFiveMinutes, toStartOfTenminutes, toStartOfFifteenMinutes and toStartOfHour

… CI failures on latest Suggestion from https://stackoverflow.com/a/71802893

* Handle distributed migrations * add extra test and update README.md

* Demonstrating breaking connection * Stop reusing broken connection * Added changelog and explanatory comment to test

Copilot

Pull Request Overview

Pull in upstream features and compatibility updates focused on distributed migrations, columnar/numpy query support, and broader Python/Django version coverage, plus tooling and docs updates.

Add distributed migrations support (including MigrationRecorder patches and tests) and HAProxy setup for load-balanced clusters.
Enable columnar and NumPy result handling via a cursor context manager; fix pool behavior when iteration is interrupted.
Expand supported/tested versions (Django 5.2, Python 3.13), update CI, tox, docs, and changelog.

Reviewed Changes

Copilot reviewed 33 out of 33 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
tox.ini	Expand env matrix, add deps tweaks and lint/format updates
tests/settings.py	Add PASSWORD for ClickHouse test connections
tests/migrations/test_loader.py	Add distributed migration tests
tests/clickhouse_functions/test_datetime.py	Add/adjust datetime function tests
tests/clickhouse_fields/tests.py	Normalize IPv6 mapped IPv4 test via ipaddress
tests/backends/tests.py	Columnar/NumPy cursor tests and iterator interruption behavior
tests/backends/clickhouse/test_driver.py	Pool size test and iterator interruption test
tests/aggregation/tests.py	Aggregator arity customization for test
pyproject.toml	Add classifiers for Django 5.x and Python 3.13
proxy-config/haproxy.cfg	New HAProxy config for load-balanced ClickHouse
docs/Configurations.md	Document max_block_size and changes
compose.yaml	Add passwords, expose HTTP ports, and HAProxy service
clickhouse_backend/patch/migrations.py	Patch MigrationRecorder and Migration apply/unapply for distributed migrations
clickhouse_backend/models/sql/compiler.py	Remove custom field_as_sql override
clickhouse_backend/models/functions/datetime.py	Add toStartOf* and toYearWeek functions
clickhouse_backend/models/fields/tuple.py	Replace deprecated is_iterable with collections.abc.Iterable
clickhouse_backend/models/fields/array.py	Replace deprecated is_iterable with collections.abc.Iterable
clickhouse_backend/models/fields/init.py	Replace deprecated is_iterable and minor checks
clickhouse_backend/models/engines.py	Replace deprecated is_iterable and normalize iterable checks
clickhouse_backend/driver/pool.py	Pool push logic to avoid reusing executing connections; arg handling
clickhouse_backend/driver/connection.py	Add columnar/NumPy execution toggles and cursor preparation
clickhouse_backend/backend/client.py	Allow passing DSN to clickhouse-client
clickhouse_backend/backend/base.py	Add distributed_migrations and max_block_size; chunked_cursor
clickhouse_backend/init.py	Bump version tuple
clickhouse-config/node[1-4]/remote-servers.xml	Add password entries for replicas
README.md	Document columnar/NumPy support and distributed migrations config
CONTRIBUTING.md	Add changelog reminder
CHANGELOG.md	Add entries for 1.3.2, 1.4.0, 1.5.0
.github/workflows/release.yml	Upgrade artifact and sigstore actions
.github/workflows/base-test.yml	Expand matrix to Python 3.13/Django 5.2 and runner pin

Comments suppressed due to low confidence (4)

clickhouse_backend/patch/migrations.py:1

[nitpick] Suggested fix for the two issues above: parameterize the cluster name and compute the replica count. Using >= 2 ensures we're only treating clusters with actual replication as replicated.

from django.apps.registry import Apps

clickhouse_backend/driver/connection.py:1

[nitpick] Proposed change using try/finally to ensure deterministic teardown even on exceptions.

import re

tests/migrations/test_loader.py:1

[nitpick] Suggested parameterized form for the query in tests.

import compileall

clickhouse_backend/patch/migrations.py:1

[nitpick] Suggested improvement to reference the actual table being created in the error message.

from django.apps.registry import Apps

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-19T10:26:33Z

clickhouse_backend/patch/migrations.py

+
+    with connection.cursor() as cursor:
+        cursor.execute(
+            f"select replica_num from system.clusters where cluster={connection.migration_cluster}"


Unsafe string interpolation in SQL and missing quoting for a string literal. Use a parameterized query to avoid SQL injection and handle string values correctly.

Suggested change

f"select replica_num from system.clusters where cluster={connection.migration_cluster}"

"select replica_num from system.clusters where cluster=%s",

[connection.migration_cluster],

Copilot · 2025-09-19T10:26:33Z

clickhouse_backend/patch/migrations.py

+            f"select replica_num from system.clusters where cluster={connection.migration_cluster}"
+        )
+        (replica_count,) = cursor.fetchone()
+    return replica_count >= 1


This query fetches a single row's replica_num instead of the replica count, so has_replicas may be incorrect. Count replicas for the cluster and compare the count rather than reading one row.

Suggested change

f"select replica_num from system.clusters where cluster={connection.migration_cluster}"

)

(replica_count,) = cursor.fetchone()

return replica_count >= 1

f"SELECT count(*) FROM system.clusters WHERE cluster = '{connection.migration_cluster}'"

)

(replica_count,) = cursor.fetchone()

return replica_count >= 2 # At least 2 replicas means there are replicas

Copilot · 2025-09-19T10:26:34Z

clickhouse_backend/driver/connection.py

+        yield self
+
+        self.use_numpy = original_use_numpy
+        self.columnar = original_columnar
+


State restoration won't run if an exception is raised inside the context. Wrap the yield in a try/finally to guarantee settings are reverted.

Suggested change

yield self

self.use_numpy = original_use_numpy

self.columnar = original_columnar

try:

yield self

finally:

self.use_numpy = original_use_numpy

self.columnar = original_columnar

Copilot · 2025-09-19T10:26:34Z

tests/migrations/test_loader.py

+    def assertMigrationExists(self, conn, name, app, deleted=False):
+        with conn.cursor() as cursor:
+            cursor.execute(
+                f"SELECT * FROM distributed_django_migrations where name = '{name}'"


Interpolating the migration name directly into SQL is unsafe. Parameterize the query to avoid SQL injection risks.

Suggested change

f"SELECT * FROM distributed_django_migrations where name = '{name}'"

"SELECT * FROM distributed_django_migrations WHERE name = %s", [name]

Copilot · 2025-09-19T10:26:34Z

clickhouse_backend/patch/migrations.py

+                editor.create_model(self.Migration)
+        except DatabaseError as exc:
+            raise MigrationSchemaMissing(
+                "Unable to create the django_migrations table (%s)" % exc


Error message hardcodes 'django_migrations' even when using a distributed table. Use the actual table name to avoid confusion.

Suggested change

"Unable to create the django_migrations table (%s)" % exc

"Unable to create the %s table (%s)" % (self.Migration._meta.db_table, exc)

Copilot · 2025-09-19T10:26:35Z

proxy-config/haproxy.cfg

+    server ch1 clickhouse-node1:9000 check
+    server ch2 clickhouse-node3:9000 check


The server hostnames don't match the docker-compose service names. In compose, services are named 'node1' and 'node3', so HAProxy should target 'node1'/'node3' instead of 'clickhouse-node1'/'clickhouse-node3'.

Copilot · 2025-09-19T10:26:35Z

proxy-config/haproxy.cfg

+    server ch1 clickhouse-node1:8123 check
+    server ch2 clickhouse-node3:8123 check


The server hostnames don't match the docker-compose service names. In compose, services are named 'node1' and 'node3', so HAProxy should target 'node1'/'node3' instead of 'clickhouse-node1'/'clickhouse-node3'.

Copilot · 2025-09-19T10:26:35Z

clickhouse_backend/__init__.py

@@ -1,5 +1,5 @@
 from clickhouse_backend.utils.version import get_version

-VERSION = (1, 3, 1, "final", 0)
+VERSION = (1, 4, 0, "final", 0)


Version tuple indicates 1.4.0 while CHANGELOG documents 1.5.0 at the top. Align the package version with the latest documented release (or adjust the changelog) to avoid release confusion.

Suggested change

VERSION = (1, 4, 0, "final", 0)

VERSION = (1, 5, 0, "final", 0)

Copilot · 2025-09-19T10:26:36Z

CHANGELOG.md

+### 1.4.0
+
+- feat: #119 Allow query results returned in columns and deserialized to `numpy` objects
+- feat: #125 Add database functions `toStartOfMinute`, `toStartOfFiveMinutes`, `toStartOfTenMinutes`, `toStartOfFifteenMinutes` and `toStartofHour`


Typo: 'toStartofHour' should be 'toStartOfHour' (capital 'O').

Suggested change

- feat: #125 Add database functions `toStartOfMinute`, `toStartOfFiveMinutes`, `toStartOfTenMinutes`, `toStartOfFifteenMinutes` and `toStartofHour`

- feat: #125 Add database functions `toStartOfMinute`, `toStartOfFiveMinutes`, `toStartOfTenMinutes`, `toStartOfFifteenMinutes` and `toStartOfHour`

Copilot · 2025-09-19T10:26:36Z

CHANGELOG.md

+
+### 1.3.2
+
+- feat(aggragation-function): add anyLast function.


Typo: 'aggragation' should be 'aggregation'.

Suggested change

- feat(aggragation-function): add anyLast function.

- feat(aggregation-function): add anyLast function.

paddycarey and others added 30 commits December 16, 2024 22:45

fix: pass DSN to clickhouse-client if configured

52ced95

feat: jayvynl#108 Queryset.iterator use clickhouse_driver.Client.exec…

574ed50

…ute_iter

chore: test for python3.13

511f97c

chore: run isort -c in code lint

858c408

docs: 1.3.2 changelog

fcc9c41

Update tox.ini

30291fa

ci: fix Error: The version '3.7' with architecture 'x64' was not foun…

9500d1b

…d for Ubuntu 24.04.

refactor base merge tree engine to check Iterable value

c61bb19

refactor model fields to check Iterable value

e8f8718

Code linting. Changelog updated

493caa4

Update base-test.yml

1facffe

test(IPv6Field): fix test for IPv6Field.

6cc97e9

Update base-test.yml

660bd77

Update tests.py

c00b4c8

Update tox.ini

3eae948

Update version to 1.3.2

b9494c3

test: Setting columnar and use_numpy for given query

af15236

feat: Allow setting of columnar and use_numpy for given_query

405bdcf

chore: Format code

b49e3b4

test: Use numpy but not columnar format to improve test coverage

ec4cffe

chore: Install numpy before running tests

2e43978

chore: Rename method to set_query_execution_args

0272d44

chore: Install pandas

cb955c4

docs: Update changelog and add documentation on set_query_execution_args

8f0bee7

Add additional clickhouse database datetime functions

f9816df

Specifically, toStartOfMinute, toStartOfFiveMinutes, toStartOfTenminutes, toStartOfFifteenMinutes and toStartOfHour

Add changelog entry

4c2d9dd

Update CHANGELOG

d576a4a

Set explicit password for default user in attempt to fix intermittent…

493b422

… CI failures on latest Suggestion from https://stackoverflow.com/a/71802893

feat: Django 5.2 Support jayvynl#122

42e23ca

Update release.yml

faee0aa

jayvynl and others added 7 commits June 5, 2025 13:04

Update release.yml

2f90a00

fix clickhouse_fields.tests.IPv6FieldTests

4966dce

Added toYearWeek functionality

df2908a

Updating changelog

c9ff3fd

Formatting and linting

59c1fe5

Support for distributed migrations (jayvynl#130)

0abe336

* Handle distributed migrations * add extra test and update README.md

Fix for simultaneous queries issue when iterating (jayvynl#133)

03b9adb

* Demonstrating breaking connection * Stop reusing broken connection * Added changelog and explanatory comment to test

Copilot AI review requested due to automatic review settings September 19, 2025 10:21

Copilot AI reviewed Sep 19, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Pulling in new changes from parent #1

Pulling in new changes from parent #1

Uh oh!

caitriona-cloudsmith commented Sep 19, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Sep 19, 2025

Uh oh!

Copilot AI Sep 19, 2025

Uh oh!

Copilot AI Sep 19, 2025

Uh oh!

Copilot AI Sep 19, 2025

Uh oh!

Copilot AI Sep 19, 2025

Uh oh!

Copilot AI Sep 19, 2025

Uh oh!

Copilot AI Sep 19, 2025

Uh oh!

Copilot AI Sep 19, 2025

Uh oh!

Copilot AI Sep 19, 2025

Uh oh!

Copilot AI Sep 19, 2025

Uh oh!

Uh oh!

	f"select replica_num from system.clusters where cluster={connection.migration_cluster}"
	"select replica_num from system.clusters where cluster=%s",
	[connection.migration_cluster],

	f"SELECT * FROM distributed_django_migrations where name = '{name}'"
	"SELECT * FROM distributed_django_migrations WHERE name = %s", [name]

	"Unable to create the django_migrations table (%s)" % exc
	"Unable to create the %s table (%s)" % (self.Migration._meta.db_table, exc)

		server ch1 clickhouse-node1:9000 check
		server ch2 clickhouse-node3:9000 check

		server ch1 clickhouse-node1:8123 check
		server ch2 clickhouse-node3:8123 check

	VERSION = (1, 4, 0, "final", 0)
	VERSION = (1, 5, 0, "final", 0)

	- feat: #125 Add database functions `toStartOfMinute`, `toStartOfFiveMinutes`, `toStartOfTenMinutes`, `toStartOfFifteenMinutes` and `toStartofHour`
	- feat: #125 Add database functions `toStartOfMinute`, `toStartOfFiveMinutes`, `toStartOfTenMinutes`, `toStartOfFifteenMinutes` and `toStartOfHour`


		### 1.3.2

		- feat(aggragation-function): add anyLast function.

	- feat(aggragation-function): add anyLast function.
	- feat(aggregation-function): add anyLast function.

Pulling in new changes from parent #1

Are you sure you want to change the base?

Pulling in new changes from parent #1

Uh oh!

Conversation

caitriona-cloudsmith commented Sep 19, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!